Method for automatically mapping light elements in an assembly of light structures

ABSTRACT

One variation of a method for automatically mapping light elements in light structures arranged in an assembly includes: defining a sequence of test frames—each specifying activation of a unique subset of light elements in the assembly—executable by the set of light structures; serving the sequence of test frames to the assembly for execution; receiving photographic test images of the assembly, each photographic test image recorded during execution of one test frame by the assembly; for each photographic test image, identifying a location of a particular light element based on a local change in light level represented in the photographic test image, the particular light element activated by the set of light structures according to a test frame during recordation of the photographic test image; and aggregating locations of light elements identified in photographic test images into a virtual map representing positions of light elements within the assembly.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 15/909,688, filed on 1 Mar. 2018, which is a continuationapplication of U.S. application Ser. No. 15/445,911, filed on 28 Feb.2017, which claims priority to U.S. Provisional Application No.62/301,581, filed on 29 Feb. 2016, and to U.S. Provisional ApplicationNo. 62/424,523, filed on 20 Nov. 2016, all of which are incorporated intheir entireties by this reference.

TECHNICAL FIELD

This invention relates generally to the field of lighting systems andmore specifically to a new and useful method for automatically mappinglight elements in an assembly of light structures in the field oflighting systems.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of a method; and

FIG. 2 is a schematic representation of a system;

FIGS. 3A, 3B, 3C, and 3D are schematic representations of one variationof the system;

FIG. 4 is a flowchart representation of one variation of the method;

FIG. 5 is a flowchart representation of one variation of the method;

FIG. 6 is a flowchart representation of one variation of the method; and

FIG. 7 is a schematic representation of one variation of the method.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. Method

As shown in FIG. 1, a method S100 for automatically mapping lightelements in an assembly of light structures includes: retrievingaddresses from light structures in the assembly in Block S110; and,based on addresses retrieved from light structures in the assembly inBlock S120, generating a sequence of test frames including a reset framespecifying activation of multiple light elements in light structuresacross the assembly, a set of synchronization frames succeeding thereset frame and specifying activation of light elements in lightstructures across the assembly, a set of pairs of blank framesinterposed between adjacent synchronization frames and specifyingdeactivation of light elements in light structures across the assembly,and a set of test frames interposed between adjacent blank frame pairs,each test frame in the set of test frames specifying activation of aunique subset of light elements in light structures within the assembly.The method S100 also includes, during a test period: transmitting thesequence of test frames to light structures in the assembly in BlockS130; and at an imaging system, capturing a sequence of images of theassembly in Block S140. Furthermore, the method S100 includes: aligningthe sequence of test frames to the sequence of images based on the resetframes, the set of synchronization frames, and values of pixels inimages in the sequence of images in Block S150; for a particular testframe in the set of test frames, determining a position of a lightelement, addressed in the particular test frame, within a particularimage, in the sequence of images, corresponding to the particular testframe based on values of pixels in the particular image in Block S160;and aggregating positions of light elements determined in images in thesequence of images into a virtual 3D map of positions and addresses oflight elements within the assembly in Block S170.

As shown in FIG. 4, one variation of the method S100 includes: readingan address from each light structure in the set of light structures inBlock S110; defining a sequence of test frames executable by the set oflight structures in Block S120, each test frame in the sequence offrames specifying activation of a unique subset of light elements acrossthe set of light structures; serving the sequence of test frames to theassembly for execution by the set of light structures in Block S130;receiving a set of photographic test images of the assembly in BlockS140, each photographic test image in the set of photographic testimages recorded by an imaging system during execution of one test frame,in the sequence of test frames, by the set of light structures; for eachphotographic test image in the set of photographic test images,identifying a location of a particular light element within a field ofview of the imaging system based on a local change in light levelrepresented in the photographic test image in Block S160, the particularlight element activated by the set of light structures according to atest frame in the sequence of test frames during recordation of thephotographic test image; and aggregating locations of light elementsidentified in the set of photographic test images into a virtual maprepresenting positions of light elements within the assembly in BlockS170.

2. Applications

Generally, the method S100 can be executed by a computer system inconjunction with an assembly of one or more light structures includingone or more light elements in order to automatically determine aposition of each light element in the assembly in real space and toconstruct a 3D virtual map of the assembly. In particular, a set oflight structures can be assembled into a unique assembly, such as on atheater or performing arts stage and connected to the computer system;the computer system can then generate a sequence of test frames in whicheach test frames specified activation of a unique subset light elements(e.g., one unique light element) in the assembly, sequentially serve thesequence of test frames to the assembly for execution while a camera ina fixed position records photographic test images of the assembly,identifies positions of light elements in the field of view of thecamera based on differences in lighted areas between photographic testimages or between photographic test images and a baseline image, andcompiles these positions of light elements in the field of view of thecamera into a virtual 3D representation of the assembly, includingrelative positions and subaddresses of each light element.

In the foregoing example, during setup the computer system can readaddresses from each light structure in the assembly in Block S110,retrieve a virtual model of each light structure based on its address inBlock S170, and align a virtual model of a light structure to lightelements—in the same light structure—identified in photographic testimages recorded by the camera to a location in the visual model in thevirtual map in Block S160. Alternatively, the computer system can servetwo sequences of test frames to the assembly, including a first sequencewhile the camera is arranged in a first location, and a second sequencewhile the camera is arranged in a second location offset from the firstlocation; the computer system can then compile detected positions oflight elements in two separate sets of photographic test images and aknown distance between the first and second locations into a 3D virtualmap of the assembly.

With a virtual map of the assembly thus constructed, the computer systemcan project a new or preexisting dynamic virtual light pattern onto thevirtual 3D map of the assembly and generate a sequence of replay framesthat, when executed by the assembly, recreate the dynamic virtual lightpattern in real space. For example, a lighting crew can assemble a setof dozens or hundreds of one-meter-cubic light structures, as shown inFIG. 2, on a stage according to a custom assembly geometry withoutregard to the orientation of these light structures and then connectthese light structures to a router without regard to order of the lightstructures. Once the assembly is complete and connected to the computersystem via the router, the computer system can execute Blocks of themethod S100 to automatically construct a virtual map of the assembly. Inthis example, a lighting engineer can then select a predefined virtuallight pattern, customize an existing virtual light pattern, code a newvirtual light pattern, or string together multiple virtual lightpatterns into a composite virtual light pattern through a user interfaceexecuting on the computer system. Once a virtual light pattern isselected, the computer system can scale, translate, and rotate thepattern in three dimensions in order to locate the virtual light patternon the virtual map and generate a sequence of replay frames, whereineach replay frame assigns a color value to each light element in theassembly according to a 3D color pattern defined by the light pattern atone instance within a duration of the predefined dynamic light pattern.

The computer system can therefore execute Blocks of the method S100 toautomatically generate a virtual map of a custom assembly of lightstructures, to automatically project a virtual dynamic 3D light patternonto the virtual map, and to automatically define a sequence of replayframes that—when executed on the assembly—recreate the virtual dynamic3D light pattern in real space without necessitating (substantial)consideration of light structure addresses or orientations duringconstruction of the assembly or manual configuration of the virtual mapwithin the user interface.

As shown in FIG. 2, the method S100 can be implemented by a system 100including one or more light structures 110 forming an assembly of lightstructures, a computer system 190 (e.g., a remote controller), a powersupply 150, a router 160, and/or an imaging system 180 (such as acamera), etc.

3. Light Structure

In one implementation, a light structure 110 defines an open cubicstructure (a “light cube”) including: twelve linear beams 112 thatdefine the twelve edges of the light cube; eight corner elements 120that each receive three perpendicular linear beams 112 to define theeight corners of the light cube; and a single chain of light elements130 (e.g., 180 LEDs) arranged along each linear beam in the light cube,as shown in FIGS. 2, 3B, and 3C.

3.1 Light Element

In this implementation, each light element 130 can include one or morelight emitters, as shown in FIGS. 3B and 3C. For example, each lightelement 130 can include a discrete red emitter, a discrete greenemitter, and a discrete blue emitter packaged in a single light element130 housing. In this example, each red, green, and blue emitter can beindependently powered (e.g., from 0% power to 100% in 256 steps) inorder to achieve a wide range of color outputs from the light element130 (e.g., ˜16.8M colors). Furthermore, each light element 130 can beindependently addressable and can be assigned a color value independentof other light elements in the chain. As in the foregoing example,during a first frame (e.g., a frame 0.2-second in duration), a lightelement 130 in the chain can be assigned a color value [0, 0, 0], or“OFF.” In this example, during a second, subsequent frame, the lightelement 130 can be assigned a color value [255, 255, 255], or “white” atfull (gamma-corrected) brightness. Furthermore, in this example, duringa third frame, the light element 130 can be assigned a color value [0,255, 255], or “cyan” at full brightness. During a fourth frame, thelight element 130 can be assigned a color value [0, 127, 127], or “cyan”at 50% (gamma-corrected) brightness.

3.2 Local Controller

In this implementation, each light element 130 can include a localcontroller that: receives a string (or array) of color values for lightelements in the chain from a preceding light element 130 (or, for thefirst light element 130 in the chain, from the control module 140described below); strips its assigned color value from the string (orarray) of color values; passes the remaining portion of the string (orarray) of color values to a subsequent light element 130; and implementsits assigned color value, such as immediately or in response to atrigger (e.g., a clock output), for each frame during operation of thelight structure 110. For example, the local controller can set a dutycycle for each color emitter in its corresponding light element 130,such as from 0% to 100% duty in 256 increments, based on color valuesreceived from the control module 140 in order to achieve a color outputfrom the light element 130 as specified by the control module 140 forthe current frame. For a subsequent frame, the control module 140 canpass a new array of color values to the chain of light elements, andeach light element 130 can repeat the foregoing methods and techniquesto output colored light according to a color value specified by thecontrol module 140 for the subsequent frame. In a similarimplementation, the light structure 110 includes multiple discretechains of light elements, wherein each chain and/or each light element130 in each chain is independently addressable with a color value. Eachchain and/or each light element 130 in each chain can thus implement theforegoing methods and techniques to output colored light according to acolor value specified by the control module 140.

In one implementation, the computer system 190 described below generatesone array of color values per light structure 110 per replay frame,wherein each array includes one color value per light element 130 in itscorresponding light structure 110. In this implementation, each colorvalue represents both an hue and a brightness of light output by thecorresponding light element 130 during execution of the replay frame.For example, for a light structure 110 that includes a chain of 180light elements connected in series and for which each light element 130contains three color elements (e.g., a red (R), a green (G), and a blue(B) color elements), the computer system 190 can output one array in theform of [R₀, G₀, B₀, R₁, G₁, B₁, R₂, G₂, B₂, R₃, G₃, B₃, . . . R₁₇₈,G₁₇₈, B₁₇₈, R₁₇₉, G₁₇₉, B₁₇₉] to the light structure 110 per replayframe. In this implementation, when a local controller of a first lightelement 130 in the chain receives a new array from the computer system190, the local controller can: strip the first three values from thearray; set a brightness of the red component of the light element 130according to the first stripped value; set a brightness of the greencomponent of the light element 130 according to the second strippedvalue; set a brightness of the blue component of the light element 130according to the third stripped value; and pass the remainder of thearray to a next light element 130 in the chain within the lightstructure 110. Upon receipt of the revised array, a local controller ofa second light element 130 in the chain can implement similar methodsand techniques to update RGB components in its light element 130, toagain revise the array, and to pass the revised array to a next lightelement 130 in the chain; etc. until the array reaches and isimplemented by the final light element 130 in the chain.

Alternatively, the first local controller can read a color value—fromthe array—at array positions predefined for the first light element 130,implement this color value, and pass the complete array to the nextlight element 130 in the chain.

Yet alternatively, the computer system 190 described below can generatea set of three arrays of color values (e.g., red, green, and bluearrays) per light structure 110 per replay frame, wherein each set ofarrays includes one color value in one color channel (e.g., red, green,or blue) per light element 130 in its corresponding light structure 110.In this implementation, each color value represents a brightness of onecomponent of a corresponding light element 130 for the replay frame.Upon receipt of a set of arrays from the computer system 190, localcontrollers at light elements in a light structure 110 can implementmethods and techniques described above to read color values from thisset of arrays and to update components of light elements in the lightstructure 110 according to these color values.

The computer system 190 and chains of light elements and localcontrollers across an assembly of light structures can repeat thisprocess for a sequence of replay frames—such as at a preset frame rate(or “refresh rate,” such as 20 Hz, 200 Hz, or 400 Hz)—during playback ofa lighting pattern, as described below.

3.3 Control Module

The light structure 110 also includes a control module 140 connected tothe chain of light elements and configured to interface with thecomputer system 190 as shown in FIGS. 3B and 3C. In one implementation,the control module 140 is preloaded with a unique ID. In one example,the unique ID defines a MAC address (or IP address) that includes: apseudorandomly assigned or serialized UUID segment; and additional data,such as a light element 130 configuration (e.g., a number of lightelements, an offset distance between light elements), a power limit ofthe light structure 110 (e.g., 25 W, 100 W), a light structure 110configuration (e.g., point light source, cubic light structure 110,tetrahedral light structure 110), and/or a make or model of the lightstructure 110; etc. The computer system 190 or router 160 can thus readthese data from the control module 140 directly. Alternatively, thecontrol module 140 can be loaded with a pseudorandomly assigned orserialized MAC address, and any of the foregoing data can be stored in aremote database and linked to the MAC address of the light structure 110via a domain name system (DNS).

3.4 Buffer

In one implementation, a control module 140 within a light structure 110includes a buffer in local memory and writes multiple arrays receivedfrom the computer system 190—via the router 160—to this buffer. Duringoperation, the control module 140 can push an oldest array in the bufferto the first light element 130 in the chain at a regular interval, suchas at a preset frame rate (or “refresh rate”, such as 20 Hz, 200 Hz, or400 Hz) and then delete this last array from the buffer once distributedto the chain of light elements. Furthermore, once the buffer is filled,the control module 140 can delete an oldest array from the buffer uponreceipt of a new array from the computer system 190 in order to maintainsynchronicity between light structures in one assembly of lightstructures to within a maximum time spanned by the buffer.

In this implementation, the control module 140 can implement a buffer ofsize based on (e.g., directly proportional to) the frame rateimplemented by the system. For example: for a frame rate of 20 fps(e.g., a refresh rate of 20 Hz), the control module 140 can implement abuffer configured to store five arrays; for a frame rate of 100 fps(e.g., a refresh rate of 100 Hz), the control module 140 can implement abuffer configured to store ten arrays; and, for a frame rate of 400 fps(e.g., a refresh rate of 400 Hz), the control module 140 can implement abuffer configured to store fifty arrays.

3.5 Communication Port

The light structure 110 also includes a communication port andreceptacle (e.g., an Ethernet port) that receives power and controlsignals from the computer system 190, such as in the form of an array ofcolor values for each light element 130 in the light structure 110. Thecontrol module 140 can also transmit its assigned MAC address (or IPaddress or other locally-stored ID), confirmation of receipt of an array(or array) of color values, confirmation of implementation of an arrayof color values across light elements in the light structure 110, and/orany other data to the computer system 190 (e.g., via the router 160)over the communication port.

3.6 Light Structure Assembly

Multiple light structures can be constructed into an assembly of lightstructures. For example, multiple light structures can be mounted (e.g.,clamped) to a truss and/or stacked and mounted to other lightstructures.

The method S100 is described herein as interfacing with andautomatically generating a virtual map of one or more light cubes.However, the method S100 can interface with and generate a virtual mapof any other one-, two-, or 3D light structures or combinations of lightstructures of various types, such as a point light source, a linearlight structure (shown in FIG. 7), a cubic light structure 110, or atetrahedral light structure 110, etc. Each light structure 110 can alsobe foldable, such as to pack flat and nest with other like lightstructures, as shown in FIGS. 3A and 3D.

4. Computer System

Blocks of the method S100 are described herein as executed by a computersystem 190, such as a computer system 190 integrated into or external tothe assembly of light structures. For example, the computer system 190can include a laptop computer (shown in FIG. 2), a desktop computer, asmartphone, a tablet, a remote server, or any other one or combinationof local, remote, and/or distributed computing devices. Blocks of themethod S100 are described herein as executed by a remote controlexternal but local to the assembly of light structures. However, Blocksof the method S100 can be executed by any other one or more local orremote computer systems.

Generally, the computer system 190 can aggregate arrays of colors valuesaddressed to specific light structures in the assembly into a frame thatcan then be loaded into and executed by the assembly. For example, thecomputer system 190 can generate frames of a lighting sequence for thelight structure 110 assembly by mapping one or more standard lightingsequence modules onto a 3D representation of the light structure 110assembly, as described below. The mobile computing device can then feedthese frames—including arrays of color values addressed to one or morelight structures within the assembly—to the assembly in order to realizethe lighting sequence, such as during a show, act, gig, exhibit, orother performance.

However, because a group of light structures may be assembled into acustom assembly and because groups of light structures may not beidentically ordered or oriented when assembled, dismantled, andreassembled (e.g., between venues), the computer system 190 can executeBlocks of the method S100, as described below, to: automaticallygenerate a virtual 3D representation (hereinafter a “virtual map”) of anassembly of one or more light structures; and to automatically map apredefined lighting sequence template, lighting sequence module, orcomplete lighting sequence to the virtual 3D representation of the lightstructure 110 assembly. In particular, the computer system 190 cantherefore execute Blocks of the method S100 prior to implementation of alighting sequence at the light structure 110 assembly in order toautomatically determine positions and orientations of a group of lightstructures—and therefore the positions and orientations of each lightelement 130 in each light structure 110—in a light structure 110assembly. The computer system 190 can therefore execute Blocks of themethod S100 retroactively such that a group of light structures can beassembled without tedious attention to order or orientation, such that ahuman operator need not manually reprogram or modify a predefined lightsequence for a current assembly of light structures, and such that anoperator need not manually map a predefined light sequence to each lightelement 130 in each light structure 110 in the current assembly.

5. Router

As shown in FIG. 2, the system can also include a router 160 interposedbetween the mobile computing device and the assembly of lightstructures. The router 160 can be connected to each light structure 110via wired connections, such as via one category-5 cable extending fromthe communication port on a light structure 110 to a corresponding portin the router 160 for each light structure 110 in the assembly.Alternatively, all or a subset of light structures in the assembly canbe connected in serial (e.g., “daisy-chained”) and connected to a singleport in the router 160 via a wired connection. The router 160 can alsoassign an IP address (or other interface identification and locationaddressing label) to each light structure 110 in the assembly. Forexample, when an assembly of light structures is completed and firstpowered on, the router 160 can read a MAC address (or other identifier)from each light structure 110 in the assembly, assign a static IPaddress to each received MAC address, populate a DNS with MAC addressand IP address pairs, and pass a list of these IP addresses (or thecomplete DNS value pairs) to the computer system 190.

During operation, such as during an automatic mapping cycle (describedbelow) or later during execution of a lighting sequence, the router 160receives frames—including an array of color values and light structure110 identifiers (e.g., IP address) for each light structure 110 in theassembly—from the computer system 190, such as over a single wired orwireless connection. Upon receipt of a frame from the computer system190, the router 160 distributes each array of color values in the frameto its corresponding light structure 110 via a port in the router 160connected (e.g., wired) to and assigned an IP address associated withthe corresponding light structure 110; light structures in the assemblythen distribute color value assignments in their received color valuearrays to corresponding light elements, and light elements across theassembly implement their respective color values, such as immediately orin response to a next clock value, flag, or interrupt transmitted fromthe router 160. For example, the computer system 190, router 160, lightstructures, and light elements can cooperate to implement a frame rateof 5 Hz (i.e., the computer system 190, router 160, light structures,and light elements can cooperate to execute a new frame once every 0.2second).

In one implementation, the router 160 receives one packet of arrays (ora “replay frame”) from the computer system 190 per update period, suchas at a rate of 20, 200, or 400 frames per second over 2 MHz serialconnection to the computer system 190. The router 160 can then funneleach array to its corresponding light structure 110. For example, atstartup (e.g., when the router 160 is first powered on followingconnection to an assembly of light structures) the router 160 can read aMAC address (or other identifier) from each connected light structure110 and then generate a lookup table linking one MAC address to one portof the router 160 for the current light structure 110-router 160configuration based on the port address from which each MAC address wasreceived. When generating a new replay frame, the computer system 190can assign a MAC address (or IP address or other identifier) to eacharray in the replay frame based on results of the automatic mappingcycle described below. Upon receipt of a new replay frame from thecomputer system 190, the router 160 can access the lookup table andfunnel each array in the replay frame through a port associated with theMAC address of its corresponding light structure 110, such as via userdatagram protocol (UDP) packets.

6. Power Supply

In one implementation, the system also includes an external power supply150, such as a power over Ethernet (PoE) switch interposed between eachlight structure 110 (or each chain of light structures) and acorresponding port in the router 160, as shown in FIG. 2. In thisimplementation, the power supply 150 can add a DC power signal to an ACdata signal received from the router 160 and pass this combinedpower-data signal to a corresponding light structure 110.

Alternatively, the system can include one or more discrete powersupplies separately wired to light structures in the assembly. A lightstructure 110 can additionally or alternatively include an integratedpower supply, such as a rechargeable battery. In this implementation,the router 160 (or the computer system 190 directly) can wirelesslybroadcast color value arrays to light structures in the assembly, suchas over short-range wireless communication protocol.

7. Light Structure Identification

Block S110 recites retrieving addresses from light structures in theassembly. Generally, in Block S110, the computer system collects IPaddresses, MAC addresses, and/or other configuration data from eachlight structure connected to the router, as shown in FIG. 1.

In one implementation, the computer system collects IP addresses fromthe router in Block S110. In this implementation, the computer systemcan pass each IP address received from the router into a DNS to retrieveconfiguration data for each light structure in the assembly, such asincluding a total number of light elements in each light structure, ashape and size (e.g., one-meter cube) of each light structure, and amaximum power output of each light structure (e.g., 25 Watts, 100Watts). For example, the computer system can pass IP addresses receivedfrom light structures in the assembly to a remote computer system overthe Internet; the remote computer system can then query a name mappingsystem for a virtual model of each light structure and then returndigital files containing these light structures to the computer systemvia the Internet. The computer system can then extract a number,configuration, and/or subaddresses of light elements in each lightstructure from these virtual models in Block S110 and implement thesedata in Block S120 to generate a sequence of test frames for theassembly. As described below, the computer system can also implementvirtual models of the light structures to construct the virtual map ofthe assembly in Block S170 described below.

Alternatively, a light structure can store its configuration data inlocal memory and can transmit these data upon request. For example, uponreceipt of an IP address from a light structure at the router, thecomputer system can query the light structure at this IP address for itsconfiguration and then store data subsequently returned by the lightstructure (e.g., a number, configuration, and/or subaddresses of lightelements in the light structure) with this IP address in Block S110

In another implementation, the computer system can receive uniqueidentifiers—encoded with light structure configuration data—from eachlight structure in the assembly. For example, once light structures areconnected to the router and the computer system is connected to therouter, the computer system can query each port of the router for aunique ID. The computer system can then extract configuration data—suchas number, configuration, and subaddresses of light elements—of eachlight structure from its unique identifier and associate theseconfiguration data with the corresponding port in the router. However,the computer system can implement any other method or technique tocollect an identifier and configuration data for each light structure inthe assembly.

8. Test Frame Sequence Generation

Block S120 of the method S100 recites, based on addresses retrieved fromlight structures in the assembly, generating a sequence of test frames.Generally, in Block S120, the computer system (or any other remotecomputer system) generates a sequence of test frames that can beexecuted by the assembly of light structures to selectively activate anddeactivate light elements across the assembly. An imaging system 190(e.g., a camera distinct from or integrated into the computer system)can capture images of the assembly in Block S140 while the assemblyexecutes the sequence of test frames, and the computer system (or otherremote computer system) can manipulate and compile these images into avirtual 3D map of real positions of each addressed light element withineach light structure within the assembly in Block S170.

In one implementation, the computer system generates a sequence of testframes including: a reset frame specifying activation of multiple (e.g.,all) light elements in light structures (e.g., all light structures) inthe assembly; a set of synchronization frames succeeding the reset frameand specifying activation of light elements (e.g., all light elements)in multiple (e.g., all) light structures in the assembly; a set of pairsof blank frames interposed between adjacent synchronization frames andspecifying deactivation of light elements (e.g., all light elements) inlight structures (e.g., all light structures) in the assembly; and a setof test frames interposed between adjacent blank frame pairs, each testframe in the set of test frames specifying activation of a unique subsetof light elements (e.g., a single light element) in light structures(e.g., a single light structure) within the assembly. For example, thecomputer system generates a sequence of test frames that, when executedby the assembly of light structures: turns all light elements in alllight structures in the assembly off; turns all light elements in alllight structures in the assembly on in the reset frame to indicate thebeginning of the test sequence; turns all light elements in all lightstructures in the assembly off; and then, for each light element in theassembly, turns the light element on while keeping all other lightelements off in the test frame, turns all light elements off in a firstblank frame, turns all light elements on in a synchronization frame toprovide a reference synchronization marker, and then turns all lightelements off in a second blank frame.

In the foregoing implementation, the reset frame defines a start of thetest sequence that can later be optically identified in the sequence ofimages—captured by the imaging system during execution of the testsequence—such as by counting a number of bright (e.g., white) pixels ineach image and comparing adjacent images in the sequence of images. Inparticular, the computer system can insert, at the beginning of the testsequence, a reset frame that defines activation of a pattern of lightelements (e.g., all light elements) in the assembly and that, whenexecuted by light structures in the assembly, can be identified in animage of the assembly to synchronize the beginning of the test sequenceto the sequence of images in Block S150. The computer system (or otherremote computer system) can also compare an image of the assemblyexecuting the reset frame to an image of the assembly executing a blankframe to generate an image mask that, when applied to an image in thesequence of images, rejects a region in the image that does notcorrespond to the assembly in Block S162, as described below. In oneexample in which each light structure in the assembly includes 180 RGBlight elements, the computer system: defines a color value arrayincluding a sequence of 180 bright white (e.g., [255, 255, 255]) colorvalues; and compiles a set of such color value arrays—including one suchcolor value array per light structure in the assembly—to generate thereset frame.

The computer system can set an extended duration for the reset frame.For example, the imaging system can be configured to capture images at aframe rate of 120 Hz, and each light structure can be configured to loadand implement a new frame at a rate of 30 Hz. During an automaticmapping cycle (and during other operating periods), the computer systemcan write a new frame to the light structure assembly at a rate of 30 Hzin Block S130 such that the imaging system can capture at least twoimages of the assembly executing each test frame in the sequence of testframes in Block S140. In this example, the computer system can specifyan extended delay between execution of the reset frame and execution ofa subsequent blank frame, such as a delay of two seconds, in the testframe sequence. Alternatively, the computer system can insert 60identical reset frames into the test frame sequence to achieve a similarreset period at the assembly during execution of the test framesequence.

In this implementation, the computer system appends the reset frame(s)with a light element test module for each light element in each lightstructure in the assembly. In particular, for each light element in theassembly, the computer system can insert a light element test modulethat specifies: a blank frame in which all light elements in theassembly are deactivated; a test frame in which a single correspondinglight element is activated; a blank frame; and a synchronization framein which all light elements in the assembly are activated (i.e., likethe reset frame). In one example, the computer system sums a totalnumber of light elements in each light structure in the assembly basedon configuration data retrieved for each light structure, as describedabove, to determine a total number of light element test modules for thetest frame sequence. In this example, the computer system then assignsan address (e.g., an IP address) of a particular light structure and anaddress of a particular light element in the particular light structureto each light element test module. For each light element test module,the computer system then: generates a unique color value array—for theassigned light structure—specifying activation of the assigned lightelement and specifying deactivation of all other light elements in theassigned light structure; generates color value arrays specifyingdeactivation of all light elements for all other light structures in theassembly; and compiles the unique color value array for the assignedlight structure and color values arrays of all light structures in theassembly into a test frame for the light element test module. Like thereset frame, the computer system can generate a generic synchronizationframe by compiling color value arrays specifying activation of (e.g., a[255, 255, 255] color value for) all light elements in all lightstructures in the assembly and can populate each light element testmodule with a copy of this generic synchronization frame. Similarly, thecomputer system can generate a generic blank frame by compiling colorvalue arrays specifying deactivation of (e.g., a [0, 0, 0] color valuefor) all light elements in all light structures in the assembly and canthen populate each light element test module with a copy of this genericsynchronization frame.

As described above, the computer system can define durations of eachblank frame, test frame, and synchronization frame in the test framesequence, such as by defined delays between blank frames, test frames,and synchronization frames in the test frame sequence. For example, thecomputer system can define 0.5-second delays when switching from a blankframe to a test frame, from a test frame to a blank frame, from a blankframe to a synchronization frame, and from a synchronization frame to ablank frame within a light element test module in order to ensuresufficient time for the imaging system to capture at least two images ofeach frame executed by light structures in the assembly. Alternatively,the computer system can insert duplicate blank frames, test frames,and/or synchronization frames in the test frame sequence in order toachieve similar frame durations.

In one variation, the computer system also generates an image masksequence and prepends the reset frame with the image mask sequence. Inone example, the computer system generates an image mask sequence byassembling a number of blank frame and synchronization frame pairssufficient to span a target mask generation duration, such as 45 blankframe and synchronization frame pairs switched at a frame rate of 30 Hzover a target mask generation duration of 30 seconds, and then insertsthe image mask sequence ahead of the reset frame in the test framesequence. The imaging system can later capture images of the assemblyexecuting the image mask sequence, and the computer system (or otherremote computer system) can generate an image mask for images capturedafter execution of the reset frame based on the images captured duringexecution of the image mask sequence in Block S162, as described below.

Furthermore, the computer system can insert a second reset frame intothe test frame sequence following a last blank frame or synchronizationframe in order to indicate the end of the test frame sequence to thecomputer system (or other remote computer system) and/or to a humanoperator. The computer system can also generate an event timelineincluding event markers corresponding to the reset frame(s), blankframes, and synchronization frames and light element subaddress markerscorresponding to light elements specified as active in test frameswithin the test frame sequence.

However, the computer system can implement any other method or techniqueto specify a color value for a light element in the assembly, toaggregate color values for light elements in a light structure into acolor value array or other container; to compile color value containersfor one or more light structures into a frame, and to assemble multipleframes into a test frame sequence that can then be executed by theassembly of light structures during an automatic mapping cycle.

8. Test Frame Sequence Execution

Block S130 of the method S100 recites, during a test period,transmitting the sequence of test frames to light structures in theassembly; and Block S140 of the method S100 recites, during the testperiod, at an imaging system, capturing a sequence of images of theassembly. Generally, during an automatic mapping cycle, the computersystem feeds frames in the test frame sequence to the assembly of lightstructures in Block S130, light structures in the assembly selectivelyactivate and deactivate their light elements based on their assignedcolor value containers in each subsequent frame received from thecomputer system (e.g., via the router), and an imaging system defining afield of view including the assembly captures a sequence of images inBlock S140, as shown in FIG. 1.

In one example, a human operator places the imaging system (e.g., astandalone camera or DSLR, a smartphone) in a first location in front ofthe assembly, orients the imaging system to face the assembly such thatthe full assembly is within the field of view of the imaging system,sets the imaging system to capture a first video (i.e., a first sequenceof images at a substantially uniform frame rate) according to BlockS140, and triggers the computer system to drip-feed frames in the testframe sequence to the assembly according to Block S130. Upon completionof the test frame sequence, the human operator places the imaging systemin a second location offset from the first location (e.g., by ten tofifteen feet), again orients the imaging system to face the assemblysuch that the full assembly is within the field of view of the imagingsystem, sets the imaging system to capture a second video (i.e., asecond sequence of images at a substantially uniform frame rate), andtriggers the computer system to again drip-feed frames in the same testframe sequence to the assembly. However, the imaging system and thecomputer system can be manually or automatically triggered in any otherway.

The imaging system can then upload the first and second videos to thecomputer system or to another computer system, such as over a wired orwireless communication protocol.

9. Image Mask

As shown in FIG. 1, one variation of the method S100 includes BlockS162, which recites generating an image mask. Generally, in Block S162,the computer system processes a subset of images captured by the imagingsystem during execution of the test frame sequence (e.g., specificallyduring the image mask sequence) to identify a region in the field ofview of the imaging system that corresponds to light elements in lightstructures in the assembly and to define this region in an image mask.The computer system can then apply the image mask to other images in thesequence of images in Block S160 to discard regions in these images thatdo not correspond to light elements in the assembly.

In the implementation above in which the computer system inserts animage mask sequence in the test frame sequence ahead of the first resetframe, the computer system can: identify a set of discard images at thebeginning of the image sequence (e.g., a first 360 images spanning afirst three seconds of the image sequence); identify a reset imagecaptured during execution of the (first) reset frame by the assembly inthe sequence of images, as described below; and select a set of imagemasks between the set of discard images and the reset image. Thecomputer system can then record an accumulated difference betweenadjacent images in the set of image masks to generate the image mask.For example, the computer system can: convert each image to grayscale(e.g., 256-bit grayscale images in which a pixel value of “0”corresponds to black and in which a pixel value of “255” corresponds towhite); and then, for each pair of adjacent images in the set of imagemasks, subtract the first image in the pair from the second image in thepair to construct a first difference image and subtract the second imagein the pair from the first image in the pair to construct a seconddifference image. The computer system can then sum all first and seconddifference images from the set of image masks, normalize this sum, andconvert this normalized sum to black and white to generate the imagemask.

In this implementation, for a pair of adjacent images captured by theimaging system while light elements in the assembly are transitioningfrom deactivated to activated according to a new mask frame, the firstdifference image will contain image data (e.g., grayscale values greaterthan “0”) representative of a region in the field of view of the imagingsystem that changes in light value when the assembly transitions from ablank frame to a mask frame. Therefore, the first difference image willcontain image data representative of a region in the field of view ofthe imaging system corresponding to light elements in the assembly (andto surfaces on light structures within the assembly and to backgroundsurfaces near light elements in the assembly). However, for this pair ofadjacent images captured while light elements in the assembly aretransitioning from deactivated to activated according to a new maskframe, pixels in the second difference image may generally saturate to“0” such that summing the first difference image and the seconddifference image yields approximately the first difference imagerepresenting a region in the field of view of the imaging systemcorresponding to light elements in the assembly.

Similarly, for a pair of adjacent images captured by the imaging systemwhile light elements in the assembly are transitioning from activated todeactivated according to a new blank frame, the second difference imagewill contain image data representative of a region in the field of viewof the imaging system corresponding to light elements in the assembly.However, for this pair of adjacent images captured while light elementsin the assembly are transitioning from activated to deactivatedaccording to a new blank frame, pixels in the first difference image maygenerally saturate to “0” such that summing the first difference imageand the second difference image yields approximately the seconddifference image representing a region in the field of view of theimaging system corresponding to light elements in the assembly.

The computer system can thus sum the first and second difference imagesfrom all pairs of adjacent images captured by the imaging system duringexecution of the image mask sequence by the assembly to generate anaccumulator image containing image data corresponding to all regionswithin the field of view of the imaging system that exhibited lightchanges during the image mask sequence. The computer system can:normalize the accumulator image by dividing grayscale values in theaccumulator image by the number of difference images compiled into theaccumulator image; apply a static or dynamic grayscale threshold (e.g.,127 in a grayscale range of 0 for black to 255 for white) to convertpixels in the normalized accumulator image of grayscale value less thanor equal to the grayscale threshold to a rejection value (e.g., “−255”)and to convert pixels in the normalized accumulator image of grayscalevalues greater less than the grayscale threshold to a pass value (e.g.,“0”). The computer system can thus define the image mask from rejectionpixels and pass pixels in the normalized accumulator image in BlockS162.

The computer system (or other remote computer system) can then convertremaining images in the sequence of images (e.g., images between thefirst and second reset images) to grayscale and add (or subtract) theimage mask to each remaining image in order to remove regions in theseimages not corresponding to light elements in the assembly, surfaces onlight structures within the assembly, or background surfaces near lightelements in the assembly from these images. The computer system canprocess these masked or “filtered” images in Blocks S150 and S160, asdescribed below.

10. Event Alignment

Block S150 of the method S100 recites aligning the sequence of testframes to the sequence of images based on the reset frames, the set ofsynchronization frames, and values of pixels in images in the sequenceof images. Generally, in Block S150, the computer system synchronizesthe sequence of images to the test frame sequence to enable subsequentcorrelation of a detected light element in an image in the imagesequence to light element subaddress specified as active in a frameexecuted by the assembly when the image was recorded, as shown in FIG.1.

In one implementation, the computer system converts images in thesequence of images to grayscale, applies the image mask to the grayscaleimages as described above, and then counts a number of pixels containinga grayscale value greater than a threshold grayscale value (e.g., “lightpixels” containing a grayscale value greater than 200 in a grayscalerange of 0 for black to 255 for white) in each image. The computersystem then flags each image that contains more than a threshold numberof light pixels as a synchronization (or reset) image captured at a timethat a synchronization (or reset) frame is executed by the assembly. Forexample, the computer system can calculate the threshold number of lightpixels as a fraction of (e.g., 70% of) pixels not masked by the imagemask. The system also identifies an extended sequence of consecutiveflagged images recorded near the beginning of the automatic mappingcycle and stores images in this sequence of consecutive flagged imagesas reset images. (The computer system can similarly identify a secondextended sequence of consecutive flagged images recorded near the end ofthe automatic mapping cycle and can store images in this second sequenceof consecutive flagged images as reset images.)

The computer system can then count the number of discrete clusters ofsynchronization images between the reset images and compare this numberto the number of discrete synchronization frames (or discrete clustersof synchronization frames) in the test frame sequence to confirm thatall synchronization events were identified in the image sequence. Iffewer than all synchronization events were identified in the imagesequence, the computer system can reduce a grayscale thresholdimplemented in Block S162 to generate the image mask, reduce a grayscalethreshold implemented in Block S150 to identify light pixels, and/orreduce a threshold number of light pixels corresponding to asynchronization in order to detect more synchronization events in theimage sequence; and vice versa.

Once the number of discrete clusters of synchronization images in theimage sequence is confirmed, the computer system can detect a test image(i.e., an image recorded while a test frame is executed by the assembly)between each adjacent cluster of synchronization frames and tag eachtest image with a subaddress of a light element activated during thecorresponding test frame. For example, for each adjacent pair ofsynchronization image clusters in the image sequence, the computersystem can: select a subset of masked grayscale images substantiallycentered between the synchronization image clusters; count a number ofpixels containing a grayscale value greater than a threshold grayscalevalue (“light pixels”) in each image in the subset of images; flag aparticular image in the subset of images that contains the greatestnumber of light pixels as a test image; and then tag the test image withthe subaddress of the light element recorded in a light elementsubaddress marker in a corresponding position in the event timelinedescribed above. The computer system can thus identify a single testimage for each test frame in the test frame sequence and tag orotherwise associate each test image with the subaddress of the (single)light element in the assembly activated when the test image was capturedin Block S150.

11. Light Element Detection

Block S160 of the method S100 recites, for a particular test frame inthe set of test frames, determining a position of a light element,addressed in the particular test frame, within a particular image, inthe sequence of images, corresponding to the particular test frame basedon values of pixels in the particular image. Generally, in Block S160,the computer system determines a position of an active light elementwithin each test image identified in the image sequence, as shown inFIG. 1.

In one implementation, for each test image, the computer system: scansthe masked grayscale test image for pixels containing grayscale valuesgreater than a threshold grayscale value (“light pixels”); countsnumbers of light pixels in contiguous blobs of light pixels in the testimage; calculates the centroid of light pixels in a particular blob oflight pixels containing the greatest number of light pixels in the testimage; and stores the centroid of the particular blob as the position ofthe light element in the test image. For example, for each test image,the computer system can store the (X, Y) pixel position of a pixel inthe test image nearest the centroid of the blob—relative to the top-leftcorner of the test image—in cells in a light element position tableassigned to the light element subaddress corresponding to the testimage. In this example, if no light pixel blob is detected in a testimage, the computer system can process an adjacent test for a lightpixel blob or can leave empty cells in the table assigned to the lightelement subaddress corresponding to the test image. In thisimplementation, the computer system can also reject blobs of lightpixels in a test image corresponding to light reflected from a surfacenear an active light element. For example, for each test image in theimage sequence, the computer system can: scan to detect light pixels inthe masked grayscale test image; count numbers of light pixels incontiguous blobs of light pixels in the test image; identify the twoblobs of light pixels containing the greatest numbers of light pixels inthe test image; select a particular blob of light pixels—from the twoblobs of light pixels—at a highest vertical position in the test imagein order to reject light reflected from a floor surface; calculate thecentroid of light pixels in the particular blob of light pixels; andstore the centroid of the particular blob as the position of the lightelement in the test image.

The computer system: can then generate a two-dimensional (2D) compositeimage representing positions of light elements in the field of view ofthe imaging system during execution of the test frame sequence fromcentroid positions (e.g., (X, Y) values) stored in the light elementposition table; and can tag each point in the composite image with thesubaddress of its corresponding light element. For the assemblyconstructed of light structures including light elements arranged alonglinear beams and tested in series during the image test sequence, thecomputer system can calculate lines of best fit (e.g., one in theX-direction and another in the Y-direction) for subsets of (X, Y) pointsrepresenting subsets of light elements in each light structure in theassembly based on light element subaddresses of light structures in theassembly and known number of light elements along linear beams in thelight structures. The computer system can then discard or flag outliersin the light element position table before transforming the table—or animage constructed from the table—into a virtual 3D map of the lightstructure in Block S170.

The computer system can repeat the foregoing Blocks of the method S100for the second sequence of images recorded by the imaging system duringexecution of the same or similar test frame sequence by the assembly inorder to generate a second composite image representing positions oflight elements in the field of view of the imaging system when arrangedat a different location relative to the assembly. However, the computersystem can implement any other method or technique to identify positionscorresponding to light elements in test images and can store thesepositions in any other way.

12. Virtual Map

Block S170 of the method S100 recites aggregating positions of lightelements and corresponding subaddresses identified in images in thesequence of images into a virtual 3D map of positions and subaddressesof light elements within the assembly. Generally, in Block S170, thecomputer system merges two or more composite images—representingpositions of light elements in fields of view of the imaging systemplaced at various locations relative to the assembly during execution ofmultiple (like or similar) test frame sequences—into a virtual 3D map ofthe light elements in light structures in the assembly, as shown in FIG.1.

In one implementation, the computer system performs three-dimensional(3D) reconstruction methods to generate a 3D point cloud from thecomposite images generated in Block S160. In particular, the computersystem can identify (X, Y) points in the first composite image thatcorrespond to the same (X, Y) points in the second composite image basedon like light element subaddress assignments, define these points asreference points, and then construct a virtual 3D representation of theassembly from the composite images based on these shared referencepoints in both composite images. In this implementation, the computersystem can also retrieve gyroscope data, accelerometer data, compassdata, and/or other motion data recorded by sensors within the imagingsystem when moved from a first position after recording the first imagesequence to a second position in preparation to record the second imagesequence, and the computer system can implement dead reckoning or othersuitable methods and techniques to determine the offset distance andrelative orientation between the first and second positions.Alternatively, the computer system can determine the offset distance andrelative orientation between the first and second positions based on GPSor other suitable data collected by the imaging system during or betweenrecordation of the image sequences. The computer system can orient thefirst and second composite images and can calculate a fundamental matrixfor the automatic mapping cycle based on the offset distance andrelative orientation between the first and second positions.

The computer system can then pass each point in the composite imagesinto the fundamental matrix to generate a 3D point cloud and can confirmviability of the fundamental matrix and alignment of the 2D compositeimages in 3D space based on proximity of reference point pairs from thecomposite images into the 3D point cloud. The computer system can selecta group of points in the 3D point cloud—corresponding to a linear beamin light structure in the assembly—and shift points in this group ofpoints into alignment with real known positions of light elements in thelinear beam of a light structure. For example, for the assembly thatincludes cubic light structures, the computer system can select groupsof points in the point cloud, each group of points corresponding to onecubic light structure, and then transform the 3D point cloud to achievenearest approximations of cubes across all groups of points in the 3Dpoint cloud. In this example, the computer system can also retrieve realoffset distances between light elements in light structures in theassembly and can scale the 3D point cloud based on these real offsetdistances between light elements. Furthermore, the computer system canmap a virtual model of a light structure, as described above, to groupsof points in the 3D point cloud to identify and discard points notintersecting or falling within a threshold distance of a light elementposition within the virtual model of the light structure. Similarly, thecomputer system can identify points missing from the point cloud andtheir corresponding light element subaddresses based on nearby lightelement subaddresses and light element positions defined in a virtualmodel of light structures inserted into the 3D point cloud, and thecomputer system can complete the 3D point cloud by inserting thesepoints and their corresponding light element subaddresses into the 3Dpoint cloud. The computer system can also shift positions of points inthe 3D point cloud to better align with light element positions definedin the virtual model of the light structure applied to the 3D pointcloud in order to improve alignment of light element represented in the3D point cloud.

However, the computer system (or other remote computer system) canimplement any other method or technique in Block S170 to generate a 3Dpoint cloud or other virtual 3D map of positions and subaddresses oflight elements within the assembly.

13. Tethered Imaging System

In one variation shown in FIG. 4, the method S100 includes: receivingaddresses from light structures in the assembly in Block S110;generating a baseline frame specifying deactivation of light elements inlight structures in the assembly in Block S120; based on addressesretrieved from light structures in the assembly, generating a sequenceof test frames succeeding the baseline frame in Block S120, each testframe in the sequence of test frames specifying activation of a uniquesubset of light elements in light structures within the assembly;sequentially serving the baseline frame followed by the sequence of testframes to the assembly for execution in Block S130; and receiving abaseline image recorded by an imaging system during execution of thebaseline frame at the assembly in Block S140. In this variation, themethod S100 also includes, for a first frame in the sequence of testframes: receiving a first photographic test image recorded by theimaging system during execution of the first test frame by the assemblyin Block S140, the first test frame specifying activation of a firstlight element in a first light structure in the assembly; andidentifying a first location of the first light element within a fieldof view of the imaging system based on a difference between the firstphotographic test image and the baseline image in Block S160. Similarly,the method S100 includes, for a second frame in the sequence of testframes: receiving a second photographic test image recorded by theimaging system during execution of the second test frame by the assemblyin Block S140, the second test frame specifying activation of a secondlight element in the first light structure; and identifying a secondlocation of the second light element within the field of view of theimaging system based on a difference between the second photographictest image and the baseline image in Block S160. The method S100 furtherincludes: retrieving a first virtual model of the first light structurebased on an address of the first light structure, the virtual modeldefining a geometry of and locations of light elements within the firstlight structure in Block S170; locating the first virtual model of thefirst light structure within a 3D virtual map of the assembly byprojecting the first virtual model of the first light structure onto thefirst location of the first light element and onto the second locationof the second light element in Block S160; and populating the 3D virtualmap with points representing locations associated with unique addressesof light elements within the assembly in Block S170.

In particular, in this variation, the method S100 can include: based onaddresses retrieved from light structures in the assembly, deactivatinglight elements across the assembly; in response to deactivation of lightelements across the assembly, capturing a first reference image of theassembly; activating light elements across the assembly; in response toactivation of light elements across the assembly, capturing a secondreference image of the assembly; and generating an image mask based on adifference between the first reference image and the second referenceimage. In this variation, the method S100 also includes: for each subsetof light elements in light structures within the assembly, activatingthe subset of light elements and, in response to activation of thesubset of light elements, capturing a test image of the assembly; foreach test image, identifying a position of an activated light elementwithin a region of the test image outside of the image mask; andaggregating positions of light elements identified in test images into avirtual 3D map of positions and subaddresses of light elements withinthe assembly.

Generally, in this variation, the computer system implements methods andtechniques similar to those described above but implements two-way wiredor wireless communication with light structures in the assembly and withthe imaging system to receive confirmation that a test frame transmittedto the assembly was executed by the assembly, to trigger the imagingsystem to capture an image of the assembly while executing a test frame,and to confirm that the imaging system captured an image of the assemblybefore loading a new test frame into the assembly. In particular, inthis variation, the computer system can: define test frames in asequence of test frames, each test frame specifying activation of singlelight element in the assembly, as described above; feed a single testframe to the assembly; trigger the imaging system to capture an image ofthe assembly once confirmation of execution of the test frame isreceived from the assembly; feed a subsequent test frame to the assemblyonly after confirmation of recordation of an image from the imagingsystem; and repeat this process for the remaining test frames in thetest frame sequence. Therefore, in this variation, the computer systemcan immediately associate each image recorded by the imaging system witha subaddress of a light element specified as active in the correspondingframe. The computer system can similarly load blank frames and maskframes into the assembly, trigger the imaging system to capturecorresponding blank images and image masks, and implement methods andtechniques described above to define an image mask based on these blankimages and image masks. The computer system can then implement methodsand techniques described above to apply the image mask to test images,identify light pixels corresponding to active elements in the testimages, and to generate a 3D point cloud or other virtual 3D map ofpositions and subaddresses of light elements within the assembly.

In this variation, the computer system can also merge positiveconfirmation of execution of test frames and their corresponding testimages to detect nonresponsive light elements and/or nonresponsive lightstructures in the assembly and to issue notifications or other promptsto test, correct, or replace nonresponsive light elements or lightstructures.

Furthermore, in this variation, the computer system and the imagingsystem can be physically coextensive, such as integrated into asmartphone, tablet, or other mobile computing device in communicationwith the router over a wired or wireless connection.

13.1 Test Frames

In this variation, the computer system can implement methods andtechniques described above to generate a sequence of test framesdefining activation of unique subsets (e.g., singular) of light elementswithin the assembly in Block S120. However, in this variation, thecomputer system can generate the set of test frames that excludes blankand reset frames. In particular, because the computer system is incommunication with the imaging system via a wired or wireless link, thecomputer system can: serve a first test frame—in the sequence of testframes—to the assembly via the router; receive confirmation from lightstructures in the assembly that the first test frame has beenimplemented; trigger the imaging system to record a first photographictest image of the assembly; and receive confirmation that the imagingsystem recorded the first photographic test image before serving asecond test frame—in the sequence of test frames—to the assembly.

In one implementation (and in implementations described above): thecomputer system defines a sequence of test frames; each test frameincludes a set of numerical arrays; each numerical array is associatedwith an address (e.g., a MAC address, an IP address) of one lightstructure in the assembly and includes a set of values; and each valuein a numerical array defines a color and intensity (or a “color value”)of one light element at one light element in the corresponding lightstructure. In the implementation described above in which each lightelement in each light structure includes a tri-color LED, each numericalarray can include one string of three values per light element in thecorresponding light structure. In this implementation, each string ofvalues corresponding to one light element can include a first valuecorresponding to brightness of a red subpixel, a second valuecorresponding to brightness of a green subpixel, and a third valuecorresponding to brightness of a blue subpixel in the light element.Upon receipt of such a numerical array, a light structure can pass thenumerical array to a first light element; the first light element canstrip the first three values from the numerical array, updateintensities of its three subpixels according to these first threevalues, and pass the remainder of the numerical array to the secondlight element in the light structure; the second light element can stripthe first three remaining values from the numerical array, updateintensities of its three subpixels according to these three values, andpass the remainder of the numerical array to the third light element inthe light structure; etc. to the last light element in the lightstructure.

Alternatively, each test frame can include one numerical array persubpixel color, such as one “red” numerical array, one “green” numericalarray, and one “blue” numerical array. Upon receipt of such a set ofnumerical arrays, a light structure can pass the red, green, and bluenumerical arrays to a first light element; the first light element canstrip the first values from each of the numerical arrays, updateintensities of its three subpixels according to these first values, andpass remainders of the numerical arrays to the second light element inthe light structure; the second light element can strip the firstremaining value from each of the numerical arrays, update intensities ofits three subpixels according to these values, and pass remainders ofthe numerical arrays to the third light element in the light structure;etc. to the last light element in the light structure.

In Block S120, the computer system generates a sequence of test frames,wherein each test frame specifies activation of a unique subset (e.g.,one) of light elements in the assembly. In one implementation, thecomputer system first identifies individually-addressable light elementsin all light structures in the assembly. For example, the computersystem can extract a number (and subaddresses) of light elements in alight structure in the assembly directly from a MAC address, IP address,or data packet received from the light structure via the router.Alternatively, the computer system can pass a MAC address, IP address,or other identifier received from the light structure to a name mappingsystem (e.g., a remote database), which can return a number (andsubaddresses) of light elements in the light structure. Similarly, thecomputer system can pass a MAC address, IP address, or other identifierreceived from the light structure to a name mapping system; the namemapping system can return a 3D virtual model of the lightstructure—including relative positions and subaddresses of lightelements in the light structure—to the computer system; and the computersystem can extract relevant information from the virtual model toidentify individually-addressable light elements in the light structure.However, the computer system can implement any other method or techniqueto characterize a light structure in the assembly, and the computersystem can repeat this process for each other light structure in theassembly.

The computer system can then generate a number of unique test framesequivalent to a number of individually-addressable light elements in theassembly. In this implementation, each test frame can specify maximumbrightness (in all subpixels) for one particular light element andspecify minimum brightness (e.g., an “off” or deactivated state) for allother light elements in the assembly.

The computer system can then define an order for these test frames suchthat, when executed by the assembly in order: a first light element in afirst light structure is activated exclusively; a second light elementin the first light structure is activated exclusively; . . . a lastlight element in the first light structure is activated exclusively; . .. a first light element in a second light structure is activatedexclusively; . . . a last light element in the second light structure isactivated exclusively; . . . a first light element in a last lightstructure is activated exclusively; . . . and a last light element inthe last light structure is activated exclusively.

Alternatively, the computer system can pseudorandomly order test framesspecifying activation of light elements within one light structure.Similarly, for the light structure that defines a cubic structure, asshown in FIGS. 3B and 3C, the computer system can order test framesspecifying activation of light elements within one light structure suchthat: a first light element along a first edge of the light structure isactivated first; followed by a second light element along a second edgeof the light structure; then a third light element along a third edge ofthe light structure; a fourth light element along a fourth edge of thelight structure; a fifth light element along a fifth edge of the lightstructure; a sixth light element along a sixth edge of the lightstructure; and then a seventh light element along the first edge of thelight structure, etc. (The computer system can similarly order testframes specifying activation of select light elements along edges of alight structure of any other form.) During execution of test framesspecifying activation of light elements within a particular lightstructure and during recordation of images of the assembly during thisprocess, the computer system can (re)calculate a location of theparticular light structure in real space and (re)calculate a confidencescore for the accuracy of this location in Block S160 following receiptof each subsequent image of the assembly recorded during activation of adifferent light element within the particular light structure. Once thecomputer system calculates a confidence score—for a location of theparticular light structure—that exceeds a threshold score, the computersystem can skip remaining test frames specifying activation of lightelements in the particular light structure and immediately transition toserving test frames specifying activation of light elements in anotherlight structure to the assembly in Block S130. Therefore, by orderingtest frames to selectively activate light elements at strategiclocations within a light structure in the assembly, the computer systemcan limit a total number of test frames necessarily executed by theassembly to yield a sufficiently accurate estimation of the reallocation of each light structure, thereby reducing a total duration ofan automatic mapping cycle.

However, the computer system can order the sequence of test framesaccording to any other function or schema in Block S120. The computersystem can also implement any other closed-loop methods and techniquesto selectively serve test frames to light structures in the assemblyuntil the location of a light structure is determined to a sufficientdegree of confidence.

In this variation, the computer system can also define a baseline framethat specifies all light elements as inactive and prepend the sequenceof images with the baseline frame. The computer system can additionallyor alternatively inject a baseline frame between test images in thesequence or append the sequence of test frames with a baseline frame.During execution of the sequence of test frames, the computer system cantrigger the imaging system to record a baseline image during executionof the baseline frame by the assembly; the computer system can thencompare the baseline image to a photographic test image recorded duringexecution of a test frame to detect a lighting change within theassembly and to associate the location of this lighting change with thelocation of a light element activated during recordation of thisphotographic test image. In one implementation in which the computersystem triggers capture of multiple baseline images during an automaticmapping cycle, the computer system can compare a photographic test imageto a most-recent baseline in order to accommodate ambient lightingchanges occurring throughout the automatic mapping cycle.

Similarly, the computer system can define a reset frame that specifiesall light elements as active and prepend the sequence of images with thereset frame. The computer system can additionally or alternativelyinject a reset frame between test images in the sequence or append thesequence of test frames with a reset frame. During execution of thesequence of test frames, the computer system can trigger the imagingsystem to record a reset image during execution of the reset frame bythe assembly; the computer system can then compare the reset image to abaseline image to generate an image mask that defines all possiblelocations of light elements in the field of view of the imaging system,as described below.

However, the computer system can define any other type, number, and/ororder of frames in the sequence of test frames.

13.2 Photographic Test Image Collection

In this variation, the computer system can also implement closed-looptechniques to selectively serve a next test frame to the assembly and totrigger the optical system to record a photographic test image of theassembly before serving a subsequent test frame to the assembly. Forexample, in Blocks S130 and S140, the computer system can: serve a firsttest frame, in the sequence of test frames, to the router fordistribution throughout the assembly; then trigger the imaging system torecord a first photographic test image of the assembly; in response toreceipt of the first photographic test image (or receipt of confirmationfrom the imaging system that an image was recorded), serve a second testframe to the router for distribution throughout the assembly; triggerthe imaging system to record a second photographic test image of theassembly; and then, in response to receipt of the second photographictest image, serve a third test frame to the router for distributionthroughout the assembly; etc.

13.3 Light Element Detection

In one implementation, the computer system analyzes photographic testimages substantially in real-time upon receipt from the imaging systemin order to construct a virtual map of the assembly throughout theautomatic mapping cycle. In this implementation, the computer system canserve a first test frame to the assembly in Block S130 and then triggerthe imaging system to record a first photographic test image. Uponreceipt of the first photographic test image from the imaging system inBlock S140, the computer system can: convert the baseline andphotographic test images to grayscale; subtract the grayscale baselineimage from the grayscale photographic test image to calculate adifference image; identify a cluster of pixels in the difference imageof value exceeding a threshold (e.g., “127”); and calculate a centroidof this cluster. The computer system can then: associate this centroidwith a 2D location of a first light element—of known subaddressspecified as active by the first test frame—in the field of view of theimaging system in Block S160. The computer system can also insert afirst point—representing and labeled with the subaddress of the firstlight element—into a 2D composite image representing locations of lightelements in the field of view of the imaging system. The computer systemcan repeat this process for subsequent test frames in the sequence topopulate the 2D composite image with points representing other lightelements in the assembly, as described above.

However, the computer system can implement any other methods andtechniques described above to transform a photographic test image (and abaseline image, etc.) into a location of a particular light elementwithin the field of view of the image sensor. The computer system canalternatively implement similar methods and techniques asynchronously totransform these images into a virtual map of the assembly followingconclusion of the automatic mapping cycle.

13.4 Image Mask

In the implement described above in which the computer system generatesa reset frame in Block S120 and serves the reset frame to the assemblyin Block S130, the computer system can generate an image mask definingknown locations of light elements in the field of view of the imagingsystem. The computer system can then scan regions of photographic testimages (or two difference images, described above) bound by the imagemask for changes in light intensity indicative of a particular lightelement in Block S160.

In one example, following receipt of a baseline image and a reset imagefrom the photographic test image in Block S160, the computer system can:convert the baseline and reset images to grayscale; subtract thegrayscale baseline image from the grayscale reset image to form adifference image mask; convert all pixels in this difference image maskof value greater than a threshold value (e.g., “127”) to “white” (e.g.,“255”) and all remaining pixels to “black” (e.g., “0”) to generate ablack and white image mask. In this example, the “white” region of theimage mask can define an “area of interest” corresponding to possiblelocations of all light elements in the assembly and locations ofadjacent surfaces (e.g., reflective floor and wall surfaces adjacent theassembly).

The computer system can then project the white region within the imagemask onto a photographic test image to bound a scan region for thephotographic test image. For example, upon receipt of a photographictest image in Block S140, the computer system can: project the area ofinterest defined by the image mask onto the photographic test image; andthen implement methods and techniques described above to selectivelyscan the area of interest projected onto the photographic test image fora bright region corresponding to a light element set as active duringrecordation of the photographic test image.

However, in this variation, the computer system can implement any othermethods or techniques, such as described above, to generate an imagemask and to apply the image mask to a region of a photographic testimage (or to a difference image) to bound a region of the photographictest image that the computer system scans for a change in lightintensity indicative of a state change of a light element in Block S160.

14. 2D-to-3D Reconstruction

In this variation, in Block S170, the computer system can transform the2D composite image—representing positions of light elements in the fieldof view of the imaging system arranged in a static location throughoutthe automatic mapping cycle—into a virtual 3D map of light elements inthe assembly. In this variation, the computer system can retrieve ageometry of a light structure in the assembly and map this geometry topoints in the composite image addressed to the same light structure inorder to approximate the 3D location and orientation of the lightstructure within the field of view of the imaging system.

In one implementation, the computer system can read a geometry directlyfrom a first light structure connected to the computer system via therouter, read a UUID from the first light structure and retrieve ageometry specific to the first light structure (e.g., in the form of avirtual model) from a database based on the UUID, or retrieve a genericor standard light structure geometry for the make and model of the firstlight structure. For example, if each light structure in the assembly isof the same type (e.g., a cubic structure with twelve edges and a stripof 180 light elements per edge), the computer system can retrieve asingle generic virtual model of all light structures in the assembly andapply this generic virtual model to calculation of a relative positionand orientation of each light structure in the assembly, as describedbelow. In this implementation, the computer system can: serve a sequenceof test frames that sequentially activate single light elements in thefirst light structure in a pseudorandom order in Block S130; trigger theimaging system to record one test image per test frame in Block S140;implement methods and techniques described above to identify a positionof a discrete light element in the field of view of the imaging systemin each subsequent test image in Block S160; and populate a compositeimage with a 2D position of a light element—and itssubaddress—identified in each subsequent test image in Block S160. Thecomputer system can then calculate a 3D location and orientation of thegeometry of the light structure that best fits points in the compositeimage, such as in real-time following addition of each subsequent pointto the composite image in Block S160.

In Block S170, the computer system: locates a composite image within a2D plane in the virtual map of the assembly; and populates the compositeimage with a first point representing a first location of a first lightelement in a first light structure detected in a first test imagereceived during an automatic mapping cycle. The computer system then:defines a first ray passing through the first point normal to the planeof the composite image; retrieves a virtual model of the first lightstructure (e.g., a 3D graphical model defining locations of lightelements and their subaddress in the first light structure); andcalculates a 3D best-fit transform that aligns (e.g., snaps, locks, orpins) a location of the first light element represented in the virtualmodel to the first point in the composite image, thereby virtuallyconstraining the virtual model of the first light structure in threedegrees of translation. Once the computer system locates a secondpoint—representing a second light element in the first lightstructure—in the composite image following receipt of a second testimage, the computer system can: define a second ray passing through thesecond point normal to the plane of the composite image; and recalculatethe 3D best-fit transform to align a location of the second lightelement represented in the virtual model to the second point in thecomposite image, thereby further constraining the virtual model of thefirst light structure in two degrees of rotation. Furthermore, once thecomputer system locates a third point—representing a third light elementin the first light structure—in the composite image, the computer systemcan: define a third ray passing through the second point normal to theplane of the composite image; and recalculate the 3D best-fit transformto align a location of the third light element represented in thevirtual model to the third point in the composite image, thereby furtherconstraining the virtual model of the first light structure in a thirddegree of rotation.

Therefore, by calculating a 3D best-fit transform that aligns locationsof light elements represented in the virtual model of the first lightstructure to points in the composite image generated from locations oflight elements detected in test images recorded during activation ofcorresponding light elements in the first light structure, the computersystem can define a function that maps the virtual model of the firstlight structure to the location and orientation of the first lightstructure in real space (e.g., relative to the imaging system). Forexample, the 3D best-fit transform can define: a lateral (e.g.,“x-axis”) translation that shifts light element locations in the virtualmodel of the first light structure into horizontal alignment with pointsin the 2D composite image; a vertical (e.g., “y-axis”) translation thatshifts light element locations in the virtual model of the first lightstructure into vertical alignment with points in the 2D composite image;a depth (e.g., “z-axis”) translation corresponding to a scale factorthat maps the virtual model of the first light structure to thedistances between points in the composite image; and pitch, yaw, androll (e.g., x-, y-, and z-axis) rotations that orient light elementlocations in the virtual model to corresponding points in the compositeimage in Block S170.

In particular, in the foregoing implementation, the computer system can:compile locations of light elements detected in test images recordedduring activation of light elements in the first light structure into a2D composite image; define one ray passing through each light elementlocation in the composite image and normal to the plane of the compositeimage; and then calculate a 3D best-fit transform that minimizesdistances between each point in the virtual model of the light structureand rays coinciding with a location of a corresponding light structurerepresented in the composite image. The computer system can repeat thisprocess for each additional light element location inserted into thecomposite image in Block S160 to recalculate the 3D best-fit transformfor the virtual model of the first light structure in Block S170.

In this implementation, following (re)calculation of the 3D best-fittransform for the virtual model of the first light structure, thecomputer system can also calculate a confidence score for accuracy ofthe 3D best-fit transform in representing the location and orientationof the first light structure in real space. For example, the computersystem can: transform the virtual model of the first light structureaccording to the 3D best-fit transform within the virtual map of theassembly; calculate nearest distances from light element locations inthe virtual map to rays passing through corresponding points in thecomposite image; and then compile these distances into a singlecomposite score representing accuracy of the 3D best-fit transform inaligning locations of light elements represented in the virtual model ofthe first light structure with points representing corresponding lightelements in the composite image. In particular, the computer system cancalculate a confidence score for the 3D best-fit transform: as afunction of (e.g., directly proportional) to distances between the lightelement locations in the virtual model and rays passing throughcorresponding points in the composite image; and as a function of (e.g.,inversely proportional to) a number of points in the composite imagesuccessfully mapped to the light element locations represented in thevirtual model of the first light structure.

Alternatively, the computer system can apply the 3D best-fit transformto the virtual model of the first light structure to locate the virtualmodel within the 3D virtual map of the assembly and project the 3Dlocation of a next light element in the virtual model onto the 2D planeof the composite image to calculate a location estimate for the nextlight element. Upon receipt of a test image recorded during execution ofa test frame specifying activation of this next light element, thecomputer system can detect a location of the next light element in thisnext test image, calculate a distance between the location estimate andthe location of the next light element detected in the next test image,and then calculate a confidence score for the 3D best-fit transform as afunction of (e.g., proportional to) this distance. However, the computersystem can implement any other method or technique to calculate aconfidence score for the 3D best-fit transform for the virtual model ofthe first light structure and can repeat this process following receiptof each subsequent test image recorded during activation of a lightelement within the light structure. (The computer system can implementsimilar methods and techniques to calculate a confidence score fordetected locations of light elements in the assembly according to Blocksof the method S100 described above.)

Once the computer system calculates a confidence score for the 3Dbest-fit transform that exceeds a minimum threshold score, the computersystem can: store the 3D best-fit transform as final for the first lightstructure; discard (e.g., withdraw, cancel) all remaining test framesspecifying activation of a light element within the first lightstructure; transition to serving test frames specifying activation oflight elements within a second light structure in the assembly; andrepeat the foregoing methods and techniques to calculate a 3D best-fittransform for a virtual model of the second light structure. Thecomputer system can repeat this process for each other light structuresin the assembly until a 3D best-fit transform is calculated for avirtual model of each light structure in the assembly.

Alternatively, the computer system can: serve a complete subset of testframes specifying activation of all light elements in a light structure;collect test images recorded during execution of each of these testframes; compile locations of light elements extracted from each of thesetest images into a 2D composite image; and then implement the foregoingmethods and techniques to calculate a 3D best-fit transform thatminimizes distances between all points in the composite image andcorresponding light element locations in the virtual model of the firstlight structure, such as upon conclusion of the automatic mapping cycle.

The computer system can therefore fuse a known geometry of a lightstructure—such as defined by a virtual model of the light structure—witha sequence of 2D test images captured during activation of select lightelements in the light structures to calculate a 3D best-fit transformthat maps the known geometry of the light structure onto the positionand orientation of the light structure in real space (e.g., relative tothe imaging system).

Once the computer system maps the light structure geometry to thecomposite image (e.g., once the computer system aligns the lightstructure geometry to points in the composite image to determine the 3Dlocation and orientation of the light structure relative to the imagingsystem in real space), the computer system can repeat this process foreach other light structure in the assembly in order to calculate 3Dbest-fit transforms that locate virtual models of these other lightstructures in positions and orientations—in the virtualmap—substantially representative of the relative positions andorientations of these light structures in real space. By applying these3D best-fit transforms to virtual models of the corresponding lightstructures, the computer system can populate the virtual map with pointsrepresenting each light element in the assembly and labeled withcorresponding subaddresses.

Therefore, rather than merge locations of light elements detected in afirst set of test frames recorded while the imaging system occupies afirst position facing the assembly of light structures and in a secondset of test frames recorded while the imaging system occupies a secondposition facing the assembly of light structures, the computer systemcan merge a virtual 3D model of a light structure with locations oflight elements detected in a set of test frames recorded while theimaging system occupies a singular position facing the assembly, therebyreducing a total duration of the automatic mapping cycle and/or a needfor manual repositioning of the imaging system. However, the computersystem can implement any other method or technique in Block S170 togenerate a 3D point cloud or other virtual 3D map representative of realpositions of light elements within the assembly and their correspondingsubaddresses.

14.1 Manual Adjustment and User Interface

In this variation, the computer system can render the virtual map of theassembly within a user interface and update the user interface (e.g., inreal-time) to show points representing real locations of light elementsand/or light structures detected in test images received from theimaging system, as shown in FIGS. 5 and 7.

In one implementation, the computer system retrieves a set of virtualmodels of all light structures in the assembly in Block S170, asdescribed above. For example, the computer system can: retrieve onegeneric virtual model for each type of light structure contained in theassembly; generate one copy of the generic virtual model for eachcorresponding light structure in the assembly; and then write asubaddress of a light element in a light structure to a correspondinglight element location in a corresponding virtual model for each lightelement in the assembly in order to generate a custom virtual model foreach light structure. In this implementation, the computer system canthen render the set of (custom) virtual models within a holding areaadjacent (e.g., to the left of) the 3D virtual map represented withinthe graphical user interface. Once the computer system locates a firstvirtual model of a first light structure in the assembly within the 3Dvirtual map (e.g., by calculating a first 3D best-fit transform for thefirst virtual model at a first time), the computer system can render ananimation showing the first virtual model moving from the holding areainto a first position—representative of a detected orientation of thefirst light structure in real space—within the 3D virtual map in thegraphical user interface. Subsequently, once the computer system locatesa second virtual model of a second light structure in the assemblywithin the 3D virtual map (e.g., by calculating a second 3D best-fittransform for the second virtual model at a second time), the computersystem can similarly render an animation showing the second virtualmodel moving from the holding area into a second position—representativeof a detected orientation of the second light structure in realspace—within the 3D virtual map in the graphical user interface. Thecomputer system can repeat process for each other light structure in theassembly.

However, if the computer system is unable to identify light elements intest images recorded during execution of test frames specifyingactivation of light elements in a particular light structure such thatthe computer system is unable to detect the position and orientation ofthe particular light structure in real space, the computer system canpreserve a virtual model of the particular light structure in theholding area in the user interface. The computer system can similarlypreserve a virtual model of a particular light structure in the holdingarea in the user interface if a confidence score for the calculatedlocation of the particular light structure (e.g., a confidence score fora 3D best-fit transform for the particular light structure) remainsbelow a threshold confidence score. In this implementation, the computersystem can prompt a user to manually locate the virtual model of theparticular light structure within the virtual map, such as uponconclusion of the automatic mapping cycle. In an example shown in FIG.5, if the computer system has failed to automatically confirm thelocation of a particular light structure upon conclusion of theautomatic mapping cycle, the computer system can: render a notificationwithin the user interface prompting the user to manually move thevirtual model of the particular light structure into its position withinthe virtual map of the assembly; generate a manipulation framespecifying activation of light elements in the particular lightstructure and deactivation of all other light elements in the assembly;update the virtual model of the particular light structure to representthe same pattern and/or color of active light elements currentlyexecuted by the particular light structure; and translate (e.g.,translate horizontally, translate vertically, and scale) and rotate thevirtual model of the particular light structure within the virtual mapaccording to inputs (e.g., a cursor drag input) into the user interface,thereby enabling the user to manually reposition the virtual model ofthe particular light structure within the virtual map. In this example,the computer system can generate the manipulation frame that specifiesactivation of light elements in blue at a first corner of the particularlight structure, activation of light elements in red at a second cornerof the particular light structure furthest from the first corner, and asubstantially smooth color transition therebetween. By updating thevirtual model of the particular light structure within the userinterface accordingly, the computer system can enable the user tovisually distinguish the orientation of the particular light structurerelative to the assembly in real space and the virtual model of theparticular light structure within the virtual map, thereby enabling theuser to both locate and orient the virtual model of the particular lightstructure properly within the virtual map.

14.2 Interference

As the computer system locates virtual models of light structures in thevirtual map of the assembly (or locates other points representing lightelements within the virtual map), the computer system can check thevirtual models for interference (e.g., virtual intersections notpossible in real space). For example, if the computer system detects anintersection between two virtual models of two light structures in thevirtual map that is not possible in real space, the computer system can:generate manual check frames that specify activation of light elementsin these two light structures (e.g., as in a manipulation framedescribed above); serve the check frame to the assembly for execution;and serve a prompt at the user interface to confirm or correct locationsof these virtual models in the virtual map. The computer system can thenimplement methods and techniques described above to adjust the positionsof the corresponding virtual models in the virtual map based on inputs(e.g., drag inputs) entered by the user into the user interface.

The computer system can implement similar methods and techniques inresponse to detection of other anomalies in the virtual map. Forexample, if the computer system calculates a location of a particularlight structure that is significantly removed from all or most otherlight structures in the assembly, the computer system can implementsimilar methods and techniques to prompt a user to confirm the locationof this particular light structure or to manually move the virtual modelof the particular light structure to its correct location within thevirtual map.

In this variation, the computer system can also update a 3D best-fittransform for a virtual model of a light structure responsive to manualadjustments to the position of the virtual model entered into the userinterface.

15. Pixel Colors

In one variation of the method S100, the computer system generatesframes specifying activation of light elements in various colors (e.g.,other than a color value of [255, 255, 255], or “bright white”).

In one implementation, the computer system generates test frames thateach specify activation of one light element in one color from a set ofoptically-discernible colors. For example, the computer system can storea set of seven optically-discernible color values including [0, 0, 255],[0, 255, 0], [255, 0, 0], [0, 255, 255], [255, 0, 255], [255, 255, 0],[255, 255, 255]. In this example, the computer system can tag orotherwise associate each test frame with both an address and a colorvalue. Thus, when identifying a duster of light pixels corresponding toan active light element in a test image, the computer system can selecta bandpass filter that passes the color value specified in thecorresponding test frame but rejects the remaining color values in theset of optically-discernible color values, and the computer system thenapplies this color filter to the test image prior to converting theimage to grayscale. The computer system can thus generate test framesthat specify activation of light elements in various color values andcan confirm that a cluster of light pixels in a test image correspondsto a particular light element addressed as active in a particular testframe based on the color of the light pixel cluster.

Furthermore, in this variation, the computer system can generate testframes that specify multiple light elements in the assembly as active atone color value from a set optically-discernible color values, and thecomputer system can correlate clusters of light pixels in acorresponding test image with subaddresses of corresponding lightelements based on color values of these light pixel clusters. Forexample, in each test frame, the computer system can set three lightelements in the assembly as active, including a first light element in afirst light structure at a color value of [0, 0, 255], a second lightelement in a second light structure at a color value of [0, 255, 0], anda third light element in a third light structure at a color value of[255, 0, 0]. The computer system can then apply three separate masks tothe corresponding test image to detect a cluster of pixels in the testimage corresponding to the first active light element, to detect acluster of pixels in the test image corresponding to the second activelight element, and to detect a duster of pixels in the test imagecorresponding to the third active light element, as described above. Thecomputer system can therefore generate test frames that specifysimultaneous activation of light elements in different color values inorder to test a group of light elements in an assembly over a shortenedperiod of time.

Therefore, in this implementation, the computer system can: generate afirst test frame specifying activation of a first light element in afirst light structure at full brightness in a first color space (e.g.,red) and specify activation of a second light element in a second lightstructure at full brightness in a second color space (e.g., green) inBlock S120; identify a first location of the first light element withinthe field of view of the imaging system in Block S160 based on adifference between the baseline image and the first color space of afirst photographic test image received during execution of the firsttest frame; and identify a second location of the second light elementwithin the field of view of the imaging system in Block S160 based on adifference between the baseline image and the second color space of thefirst photographic test image. The computer system can thereforeidentify multiple unique light elements within a single photographictest image by leveraging multiple color spaces supported by both lightelements in the light structures and the imaging system, therebyreducing a total duration of the automatic mapping routine.

Alternatively, the computer system can: generate test frames thatspecify activation of unique sets of light elements at the same colorvalue (e.g., [255, 255, 255], or “bright white”), wherein each lightelement is specified as active at least twice in the test framesequence; detect clusters of light pixels corresponding to multipleactive light elements in each test image, as described above; and then,for each light element, assign a subaddress of a particular lightelement to a particular duster of light pixels in a test image inresponse to a second test image—corresponding to a second test framealso specifying activation of the particular light element—including acluster of light pixels in substantially the same location as thecluster of light pixels in the first test image. In this implementation,the computer system can then implement methods and techniques describedabove to generate a 3D point cloud or other virtual 3D map of positionsand subaddresses of light elements within the assembly.

However, the computer system can generate test frames specifyingactivation of any other number or combination of light elements in anyother one or more colors.

16. Light Patterns

As shown in FIG. 6, one variation of the method S100 includes BlockS180, which recites: mapping a predefined dynamic light pattern onto the3D virtual map; generating a sequence of replay frames, each replayframe in the sequence of replay frames assigning a color value to eachlight element in the assembly according to a color pattern defined bythe predefined dynamic light pattern at one instance within a durationof the predefined dynamic light pattern; and serving the sequence ofreplay frames to the assembly for execution. Generally, in Block S180,the computer system can map predefined virtual light patterns onto thevirtual representation of the assembly in order to generate replayframes specifying activation of particular light elements in particularcolors; the computer system can then load these replay frames into theassembly for execution by the light structures in order to recreate thispredefined virtual light pattern in real space.

In one implementation, for each video frame in a 3D video, the remotecontroller can: project a video frame onto the virtual map of lightelements in the assembly to generate a replay frame for the assembly;and then load these replay frames into the assembly in order to replay aform of the 3D video in real space. In particular, by mapping a videoframe from a light pattern onto the virtual map of light elements, thecomputer system can calculate a color value for each light element ineach light structure in the light structure assembly. The computersystem can store each color value for each light element in an arrayassigned to one light structure and group a set of arrays—for anassembly of light structures—into one replay frame corresponding to thevideo frame. When each array in the replay frame is (substantiallysimultaneously) executed by its assigned light structure, the lightstructure assembly can physically represent the virtual video frame. Thecomputer system can repeat this process for each video frame in thelight pattern to generate one “RGB” array per light structure in thelight structure assembly per replay frame of the light pattern.

In a similar implementation shown in FIG. 6, the computer systemaccesses a parameterized dynamic light pattern—selected by a userthrough the user interface—defining a virtual volume of color thatchanges in color value and intensity as a function of time. The computersystem then: scales the virtual volume of the parameterized dynamiclight pattern to the height, width, and depth of the point cloud oflight elements in the virtual map of the assembly; and associates eachpoint in the virtual map with a discrete virtual subvolume in thevirtual volume of the parameterized dynamic light pattern. The computersystem can also: record various parameters entered into the userinterface, such as speed and intensity; adjust a speed of parameterizeddynamic light patterns according to the speed selection; scale colorvalues in each color space in the parameterized dynamic light patternaccording to the intensity selection; and/or adjust the parameterizeddynamic light pattern according to any other parameter. The computersystem can then generate a replay frame for one instance of theparameterized dynamic light pattern by writing a color value of asubvolume—at the instance—to its corresponding point for each point inthe virtual map. The computer system can implement a similar process togenerate a replay frame for other instance within the parameterizeddynamic light pattern, such as at a preset frame rate of 20 Hz. By thenserving these replay frames to the assembly in sequence in Block S180,the computer system can recreate the parameterized dynamic light patternin real space. In this implementation, the computer system can generateand serve replay frames to the assembly in real-time; the computersystem can also adjust the parameterized dynamic light pattern inreal-time based on parameters adjusted at the user interface.

In another implementation, the computer system generates a tablecontaining subaddresses and real (X, Y, Z) positions of light elementsin the assembly and passes the table into a user-driven replay framegenerator (e.g., a replay frame generator application executing on thecomputer system) to reconstruct the assembly virtually within a userinterface. The user can then select points within the user interface toactivate or deactivate to create replay frames specifying activation anddeactivation of corresponding light element subaddresses in theassembly, and the computer system can feed these replay frames to theassembly asynchronously or in real-time. In this implementation, theuser-driven replay frame generator can include predefined lightpatterns, such as rainbow frames, horizontal and vertical light linesweep modules (i.e., a sequence of multiple replay frames), pulsinglight modules, etc., and the user-driven replay frame generator canscale and transform a predefined light pattern selected by the user ontothe virtual representation of the assembly in order to generate customreplay frames—that mimic the selected light pattern—for the assembly.The user-driven replay frame generator can also speed up and slow downpredefined light pattern modules selected by the user, merge or smoothtransitions between adjacent light pattern modules, etc. to create asequence of replay frames for the assembly.

In a similar implementation, the computer system can generate a graph oflight paths through the assembly. For each example, the computer systemcan generate a graph of nearest light elements within the assembly,receive a selection for a starting light element subaddress in theassembly, receive a selection for an ending light element subaddress inthe assembly, and then create a sequence of replay frames that, whenexecuted by the assembly, “snake” a small cluster of adjacent activelight elements from the starting light element to the ending lightelement.

However, the computer system can implement any other method or techniqueto generate replay frames customized to the assembly (i.e., customizedto the substantially unique arrangement of light structures in theassembly) based on the virtual representation of the assembly generatedin Block S170.

15.1 Interpolation

In one implementation in which a lighting program executed by thecomputer system outputs replay frames at a frame rate less than a targetframe rate of the system (e.g., 10 fps and 20 fps, respectively), thecomputer system can interpolate between two replay frames output by thelighting program to generate intermediate frames containing intermediatearrays; the computer system can serve both replay and intermediateframes to the assembly of light structures—via the router—in order toachieve the target frame rate and smoother transitions between replayframes.

For example, the computer system can calculate an intermediate array—inreal-time or asynchronously—by interpolating between an array in acurrent replay frame sent to a light structure and an array in a lastreplay frame sent to the light structure and serve this intermediatearray to the light structure before sending an array in a next replayframe to the light structure, thereby doubling the effective frame rateof the system. In this example, the computer system can implement linearinterpolation or other interpolation techniques. The computer system canalso interpolate between two (or more) replay frames to generatemultiple intermediate frames between two replay frames, such as nineteenintermediate frames between each consecutive pair of replay frames toachieve a target frame rate of 400 fps (i.e., a refresh rate of 400 Hz)given a lighting program outputting replay frames at 20 fps.

15.2 Dithering

In another implementation, the computer system can inject noise intocolor values in arrays across multiple replay (and intermediate) frames,which, when executed by an assembly of light structures, may beperceived by a user as greater color depth in light elements acrossthese light structures. In particular, the computer system canintentionally inject noise into arrays in a series of replay frames inorder to achieve perception of greater color depth and/or perception ofa greater number of colors for onlookers.

In one example, the computer system can inject or apply a sinusoidalnoise signal—at a preset noise magnitude and frequency—to color valuesstored in arrays within replay (and intermediate) frames across asequence of replay frames, such as a common sinusoidal noise signal toeach color component or unique sinusoidal noise signals at variousmagnitudes, frequencies, and/or phases to the color components of lightelements in the light structures. Alternatively, the computer system canimplement a pseudorandom noise generator and noise magnitude cap (e.g.,a magnitude cap of “10” for an eight-bit color value) to (severely orsubtly) adjust color values in arrays across a sequence of replay framesserved to the assembly light structures. By executing arrays injectedwith noise, light structures in a light structure assembly can outputlight that “flickers” through colors around original colors specified inthe original light pattern, which may be perceived by an onlooker asgreater color depth.

However, the computer system can implement any other method or techniqueto intentionally inject noise into arrays served to light structures.

15.3 Low Color Brightnesses

In another implementation, the computer system can implement and adjustcolor values across a sequence of arrays served to a light structure inorder to achieve greater perceived brightness resolution at low colorbrightnesses.

In one example, the computer system can generate and output arrayscontaining 8-bit color values. In this example, upon receipt of a newarray from the computer system, a local controller connected to aparticular light element can read an assigned color value of [25, 25,25] from this new array and interpret this color value as a callout for10% brightness (i.e., “ 25/255”) at each of the red, green, and bluecomponents in its corresponding light element. The local controller inthe light structure can then pulse-width-modulate (“PWM”) power signalssupplied to these red, green, and blue components at 10% duty, such asat a static PWM rate of 2 kHz. In this example, for the system thatimplements a frame rate of 200 Hz, the local controller will update astate of the red, green, and blue components in its corresponding lightelements ten times before a next array is received from the computersystem. To achieve 10% duty at the red, green, and blue components inthe particular light element, the local controller can thus activatethese red, green, and blue components once in the ten PWM cyclespreceding receipt of a next array from the computer system, which may beperceived by an onlooker as “white” light at approximately 10%brightness for this replay frame.

However, if the local controller reads a color value of [5, 5, 5] in anext array received from the computer system, the local controller caninterpret this color value as 2% brightness (i.e., “ 5/255”) for each ofthe red, green, and blue components in its corresponding light element.By implementing this [5, 5, 5] color value directly to achieve 2% dutyat the red, green, and blue components in the particular light elementat the frame rate set of 200 Hz, the local controller may activate thesered, green, and blue components once or never in ten PWM cycles before anext array is received from the computer system. The local controllermay therefore be unable to reliably achieve such low color brightnesstargets by executing a single array specifying such low colorbrightness—that is, for color values less than “25” or 10% brightnessfor a PWM rate of 2 kHz and a frame rate of 200 Hz.

Therefore, rather than serve arrays containing [5, 5, 5] color valuesfor the particular light element to achieve perception of white light at2% brightness at the particular light element, the computer system cangenerate and serve a sequence of arrays that—in aggregate—yield a colorbrightness specified in a light pattern, as described above. Forexample, to achieve 2% color brightness at a particular light elementover a period of 500 ms given a frame rate of 200 Hz and a PWM frequencyof 2 kHz, the computer system can output a sequence of 100 arrays to thelight structure containing the particular light element. In thisexample, the computer system can pair a set of four arrays containing a[0, 0, 0] (or “full off”) color value for the particular light elementwith one array containing a [25, 25, 25] (or “10% brightness”) colorvalue for the particular light element; these 100 arrays can includetwenty such sequences. When the particular light executes color valuesin these 100 arrays in sequence over a period of 500 ms, an onlooker mayperceive “white” light at approximately 2% brightness from theparticular light element over this period of time.

The computer system can therefore implement second order-PWM techniquesto modulate a color value assigned to a particular light element over asequence of arrays in order to more accurately achieve a low colorbrightness at the particular light element based on a frame rate of thesystem and a PWM frequency implemented by local controllers within thelight structures.

The systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive a computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated bycomputer-executable components integrated with apparatuses and networksof the type described above. The computer-readable medium can be storedon any suitable computer readable media such as RAMs, ROMs, flashmemory, EEPROMs, optical devices (CD or DVD), hard drives, floppydrives, or any suitable device. The computer-executable component can bea processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

I claim:
 1. A method for automatically mapping light elements in a setof light structures arranged in an assembly, the method comprising:serving a sequence of test frames to the assembly for execution by theset of light structures, each test frame in the sequence of framesexecutable by the set of light structures and specifying activation of asubset of light elements within the set of light structures; receiving aset of optical test images of the assembly, each optical test image inthe set of optical test images recorded by an imaging system duringexecution of one test frame, in the sequence of test frames, by the setof light structures; for each optical test image in the set of opticaltest images, detecting a location of a particular light element within afield of view of the imaging system based on a local change in lightlevel represented in the optical test image, the particular lightelement activated within the set of light structures according to a testframe in the sequence of test frames executed by the assembly duringrecordation of the optical test image; and aggregating locations oflight elements detected in the set of optical test images into a virtualmap representing positions of light elements within the assembly.